Papers with video representation
Modality Alignment between Deep Representations for Effective Video-and-Language Learning (2022.lrec-1)
Copied to clipboard
| Challenge: | Existing Video-and-Language models do not take into account the different characteristics of video and text representations. |
| Approach: | They propose a method that exploits Centered Kernel Alignment (CKA) to enhance cross-modality attention by combining multiple modalities. |
| Outcome: | The proposed method outperforms conventional multi-modal methods significantly on video QA tasks with +3.57% accuracy increment compared to the baseline in a popular benchmark dataset. |
Dense Procedure Captioning in Narrated Instructional Videos (P19-1)
Copied to clipboard
| Challenge: | Existing models for video dense captioning learn video segments and generate captions without considering transcripts. |
| Approach: | They propose a model to generate procedure captions from narrated instructional videos . they extract procedures by a cross-modality module and generate captions by encoding video frames and transcripts within each extracted procedure. |
| Outcome: | The proposed model can extract procedures from narrated instructional videos and generate procedure captions by encoding video frames and transcripts. |